Details:

• Backstory: [Bug]: Inference fails if data is moved to GPU #25484
• The original issue happens because a GPU Torch tensor is being passed to OpenVINO inference
• Python API always creates a np.array before passing the data to Tensor constructors
• numpy only supports CPU memory (more info)
• Because of it, the data has to be copied to CPU memory before initializing np.array
• Because the model in the user's case is running on GPU, it has to be copied again, so the final copying is GPU -> CPU -> GPU
• This causes a significant performance loss
• The new constructor uses a pointer directly to GPU memory to create Tensor
• Using it reduces the average inference time in customer's script from 100ms to 50ms on my machine

Example usage:

image = torch.rand(128, 3, 224, 224)
image = image.to(torch.device("xpu"), memory_format=torch.channels_last)
data_ptr = image.detach().data_ptr()
ov_tensor = Tensor(data_ptr, Shape(image.shape), pt_to_ov_type_map[str(image.dtype)])

To be discussed

• Since the new ctor does not take array as an argument but a pointer to it, the reference count for array is not incremented (see the new test and the one above it). The ctor can't take the array as argument and retrieve its pointer in the binding, because it would interfere with other Tensor ctor overloads.
• Reference count incrementation can be forced with a pure Python wrapper, but then the Tensor would need to be created with a separate util such as tensor_from_ptr().
• We can't expand data_dispatcher.py beacuse there already is a dispatch for an int
• Is there another option?

Tickets:

• CVS-154510